Only read file metadata once in LocalFileSystem::read_ranges#595
Merged
alamb merged 1 commit intoapache:mainfrom Jan 8, 2026
Merged
Only read file metadata once in LocalFileSystem::read_ranges#595alamb merged 1 commit intoapache:mainfrom
LocalFileSystem::read_ranges#595alamb merged 1 commit intoapache:mainfrom
Conversation
5696f91 to
a53a452
Compare
LocalFileSystem::read_ranges
a53a452 to
4f3c6bb
Compare
iakshay
approved these changes
Jan 7, 2026
Contributor
Author
|
I've put a quick benchmark together: use criterion::{BenchmarkId, Criterion, Throughput, criterion_group, criterion_main};
use object_store::local::LocalFileSystem;
use object_store::path::Path;
use object_store::{ObjectStore, ObjectStoreExt};
use rand::Rng;
use std::ops::Range;
use tempfile::TempDir;
const FILE_SIZE: u64 = 64 * 1024 * 1024; // 64 MB
const RANGE_SIZE: u64 = 8 * 1024; // 8 KB ranges
fn generate_random_ranges(file_size: u64, range_size: u64, count: usize) -> Vec<Range<u64>> {
let mut rng = rand::rng();
(0..count)
.map(|_| {
let start = rng.random_range(0..file_size - range_size);
start..start + range_size
})
.collect()
}
fn bench_read_ranges(c: &mut Criterion) {
let rt = tokio::runtime::Builder::new_current_thread()
.enable_all()
.build()
.unwrap();
// Set up the test file
let temp_dir = TempDir::new().unwrap();
let store = LocalFileSystem::new_with_prefix(temp_dir.path()).unwrap();
let path = Path::from("bench_file");
// Create file with random data
let data: Vec<u8> = (0..FILE_SIZE).map(|i| (i % 256) as u8).collect();
rt.block_on(async {
store.put(&path, data.into()).await.unwrap();
});
let mut group = c.benchmark_group("read_ranges");
for num_ranges in [10, 100, 1000] {
let ranges = generate_random_ranges(FILE_SIZE, RANGE_SIZE, num_ranges);
let total_bytes = num_ranges as u64 * RANGE_SIZE;
group.throughput(Throughput::Bytes(total_bytes));
group.bench_with_input(
BenchmarkId::new("local_fs", num_ranges),
&ranges,
|b, ranges| {
b.to_async(&rt)
.iter(|| async { store.get_ranges(&path, ranges).await.unwrap() });
},
);
}
group.finish();
}
criterion_group!(benches, bench_read_ranges);
criterion_main!(benches);The improvement on my local macbook looks like 10-20%: |
kylebarron
approved these changes
Jan 7, 2026
Contributor
|
This would also be a good one to backport to I am hoping to work on that release tomorrow |
8 tasks
Contributor
Author
|
I'll open a backport PR tomorrow |
alamb
approved these changes
Jan 7, 2026
AdamGS
added a commit
to AdamGS/arrow-rs-object-store
that referenced
this pull request
Jan 8, 2026
…`LocalFileSystem::get_ranges`
alamb
added a commit
that referenced
this pull request
Jan 8, 2026
…FileSystem::get_ranges` (#596) Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Contributor
Author
|
Also happy to backport it to 0.13.2, but I can't find a branch |
Contributor
We'll make the 0.13.2 release from Though now that you mention it, maybe we want to accelerate the 0.13.2 timeline to get this fix out sooner than Feb 🤔 |
Contributor
|
I wonder, why do we even want to read the metadata before retrieving the bytes? @AdamGS
|
Contributor
Author
|
that's a very good point |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #594
Rationale for this change
As described in issue, seems wasteful to read and drop the file's metadata multiple times.
What changes are included in this PR?
Hopefully small performance improvement.
Are there any user-facing changes?
None.